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ABSTRACT 

As a preliminary to developing criteria to meet 
quality control requirements of the National Literacy Act of 1991, a 
study examined federal and state development of measures of program 
quality and performance standards for programs funded under the Job 
Training Partnership Act (JTPA), the Carl D. Perkins Vocational and 
Applied Technology Education Act, the Food Stamp Employment and 
Training Program (FSETP) , and the Job Opportunities and Basic Skills 
(JOBS) program of the Family Support Act. The study found that JTPA 
implemented performance standards for employment and training 
programs at the inception of the program that measured cost per 
entered employment, entered employment rate, and average wage at 
placement. Reacting to criticism that these standards forced programs 
to neglect services to the hard-to-place, JTPA created new standards 
in 1990 stressing job retention. The Perkins Act reauthorization of 
1990 requires states to develop performance standards that measure 
learning gains, competency or job skill attainment, placement into 
employment or other training programs, and retention in or completion 
of secondary school or its equivalent; about half the states are 
having considerable difficulty developing such standards. FSETP and 
JOBS programs are required to implement performance standards, but 
they have not yet been developed. Based on the experience of these 
other programs with performance indicators, the following lessons 
appear most relevant to adult education: (1) establish appropriate 
indicators and standards; (2) allow state and local flexibility; (3) 
develop management information systems; (4) provide states with 
technical assistance; and (5) take steps to avoid unintended effects 
of indicators. (11 references) (KC) 
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QUALITY INDICATORS FOR ADULT EDUCATION PROGRAMS: 
LESSONS LEARNED FROM OTHER PROGRAMS 



EXECUTIVE SUMMARY 



The National Literacy Act of 1991 requires the states and U.S. Department of Education 
to develop indicators of program quality to evaluate programs funded under the Adult Education 
Act. This paper informs efforts to develop these indicators by examining Federal and state 
experience in developing measures of program quality and performance standards for programs 
funded under the Job Training Partnership Act (JTPA), the Carl D. Perkins Vocational 
Education and Applied Technology Act, the Food Stamp Employment and Training Program 
(FSETP), and the Job Opportunities and Basic Skills (JOBS) program of the Family Support Act. 

JTPA implemented performance standards for employment and training programs at the 
inception of the program that measured cost per entered employment, entered employment rate 
and average wage at placement. The performance standards were criticized for emphasizing quick 
job placement at low cost and "creaming"~serving the more job-ready-of participants. National 
evaluations of the impact of performance standards verified these complaints, uncovering evidence 
of a preference for low intensity service and reduced service to hard-to-serve populations. These 
unintended effects of the performance standards were worse in service areas that lacked clear 
policy goals and emphasized cost standards. To address these problems, the cost standard was 
dropped and new performance standards were implemented in 1990 that stressed job retention. 

The 1990 reauthorization of the Perkins Act, which funds vocational education programs, 
required the states to develop within two years performance standards that measured (1) learning 
gains, (2) competency or job skill attainment, (3) placement into employment or other training 
programs, and (4) retention in or completion of secondary school or its equivalent. A recent 
study has shown that about half the states are experiencing considerable difficulty developing 
standards due to lack of experience with evaluation and measures and the diversity of vocational 
education providers and service population. Many states are considering different standards for 
different types of providers. 

Separate legislation creating FSETP and JOBS required these programs to implement 
performance standards but neither program has yet developed them. A set of standards for 
FSETP developed by a national committee were not approved by Congress due to perceived 
excessive data collection requirements on local programs. FSETP and JOBS are currently held to 
participation requirements. FSETP programs initially had a requirement to place 50 percent of 
participants in employment programs which was later reduced to 10 percent to allow programs to 
focus more on quality of placement. JOBS program participants must receive an average of 20 
hours per week of education, training or job placement activity. Participation requirements for 
both programs have been criticized as promoting low intensity services. 

Adult education programs may learn in five areas from the experience of these programs 
in developing indicators of program quality at the state and national level. 
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Indicators should be developed as part of a systematic process that considers 
program evaluation, planning and improvement. The indicators should have 
validity, be inexpensive to measure and be consistent from program to program 
across time. 

Local providers should have flexibility to define how some indicators are measured 
and to determine the numeric level of performance standards. JTPA allows local 
adjustment of standards to account for local conditions, training providers and 
service populations. Many states are also setting separate vocational education 
standards for different types of providers. The diversity of the provider network 
and participants in adult education programs argues for similar adjustment 
provisions. 

A uniform, automated management information system (MIS) allows for collection 
of a common set of data, minimizes burden and facilitates analysis of quality 
indicators and performance standards. The JTPA program uses this approach 
successfully. Tlie rejection of standards developed for FSETP due to lack of data 
collection capabilities illustrates the barriers a lack of an MIS can have on use of 
quality indicators. Adult education currently lacks good data collection capabilities 
at the Federal level and in most states. 

Technical assistance needs to be provided to states and local programs in the 
development and implementation of performance standards and indicators. There 
is ongoing interaction in the JTPA program on performance standards at all levels 
and states may use set aside funds to provide technical assistance to local 
providers. The U.S. Department of Education is currently funding technical 
assistance to states in the development of vocational education standards. Similar 
effort should be considered for adult education programs. 

The JTPA experience demonstrates that performance standards can produce 
unintended effects such as "creaming" and over-emphasis on low-intensity services. 
JTPA also has learned that these effects can be minimized when there is (1) clear 
expression of Federal policy on how standards relate to policy goals, (2) ongoing 
input and consultation from local providers on the implementation and impact of 
standards, and (3) emphasis on and provision of incentives for meeting standards 
that reflect policy-relevant goals. Setting the numeric level of the standard at a 
moderate level also will minimize negative impacts. 



QUALITY INDICATORS FOR ADULT EDUCATION PROGRAMS: 
LESSONS LEARNED FROM OTHER PROGRAMS 



Since its passage in 1966, the Adult Education Act (AEA) has been the Federal 
Government's primary mechanism for supporting services that address the problems of adult 
illiteracy. Recent amendments to the AEA contained in the National Literacy Act of 1991 (P.L. 
102-73) reaffirm the purpose of the Act encouraging the establishment of adult education and 
literacy programs that will: 

• Enhance the literacy and basic skills of adults; 

• Ensure that all adults acquire the basic skills necessary to function effectively and 
achieve the greatest possible opportunity in their work and in their lives; and 

• To strengthen and coordinate adult literacy programs. 

The AEA authorizes a number of programs, the largest of which is a state grant program 
that funds three types of activities: (1) basic literacy services for adults whose skill levels are 
below the eighth grade; (2) services designed to prepare students to obtain a high school 
equivalency diploma; and (3) English-as-a-second language (ESL) services for adults with limited 
English proficiency. 1 Local projects funded by the state grants program include school districts, 
community colleges, and community-based organizations. 

The National Literacy Act amendments require the states and Federal Government to 
establish 

. . . indicators of program quality to be used ... to determine whether [adult 
education] programs are effective, including whether such programs are 
successfully recruiting, retaining and improving the literacy skills of individuals 
served in such programs. (Section 331 (a)(2)) 

The statute requires the Department of Education (ED) to develop illustrative indicators by July 
1992. States will have until July 1993 to develop their own indicators. In developing these 
indicators, states are to consult with experts in adult education, educators, and administrators. 

An indicator of program quality is a single statistic or variable that can be used to 
determine whether a program has been successful. Program quality indicators must reflect the 
objectives of the program and consist of measurable data that can be collected by local projects. 
A numeric criterion or performance standard is sometimes set for indicators to establish a 
benchmark by which to judge program performance on the measure. To inform the development 
of program quality indicators for the AEA, this report reviews methods and measures, including 
performance standards, used by four other Federal programs that provide education and training 
for adults who lack basic literacy skills or who are economically disadvantaged: the Job Training 
Partnership Act (JTPA), the Carl D. Perkins Vocational Education and Applied Technology Act, 



1 Also authorized by the AEA are the National Workplace Literacy Program, English Literacy 
Program, National English Literacy Demonstration Programs, Adult Migrant Farmworkers and 
Immigrant Education, and National Adult Literacy Volunteer Training Program. 



the Food Stamp Employment and Training Program (FSETP), and the Family Support Act's Job 
Opportunities and Basic Skills (JOBS) program. 2 

In addition to having similar objectives and serving clients with similar characteristics, 
JTPA, the Perkins Act, FSETP, and JOBS have statutory requirements for performance 
standards. JTPA programs are required to use a system of national performance standards to 
assess participant outcomes and program effectiveness. JOBS legislation and the Perkins Act 
require their programs to implement a similar system of performance standards within the next 
several years. We draw from the experience of these programs to suggest factors to consider in 
developing model indicators of program quality for the adult education program. 3 

The Job Training Partnership Act 

The Job Training Partnership Act, enacted in 1982, is the primary source of Federal 
funding for employment and training programs for economically disadvantaged youth and adults 
and displaced workers. JTPA was designed as a partnership between state and local governments 
and private industry. Types of training available through JTPA are on-the-job training, work 
experience, job search assistance, basic education, and occupational skills. 

JTPA funds are distributed to states, which in turn designate service delivery areas (SDAs) 
to be responsible for service provided in that area. 4 SDAs consist of one or more units of local 
government which are congruent with labor market areas. They are usually formed along county 
lines or by combining contiguous counties and cities into one area. In some sparsely populated 
states, governors designate the entire state as a single SDA 

Private industry councils (PICs), composed of local business and community leaders, 
develop and manage programs within the SDA The PICs develop job training plans for the 
SDA, establish guidelines for occupational skills training, award and monitor training contracts 
with local training providers. Training agencies may be community-based organizations, non-profit 
or proprietary schools, community colleges, the local school system, or vocational education 
training agencies. 



Employment is not necessarily the goal of AEA programs, although the student's 
employability may be enhanced as a result of participation. JTPA, vocational education and JOBS 
programs provide employment-related training that is designed to allow the participant to obtain a 
job upon completion of the respective programs. 

3 Each of these programs is in a different stage of implementing performance indicators. 
JTPA is the furthest along of the three programs and offers the most real-world experiences for 
the adult education field. 

4 Under JTPA, each state receives a block grant allocation, 78 percent of which must be spent 
on training and distributed to the local level. The remaining funds are required to be setasides 
for specific purposes. For example, the six percent setaside is reserved for local technical 
assistance and incentive awards for exceeding performance standards and serving hard-to-serve 
groups. 

2 
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JTPA Performance Standards 



JTPA has established both indicators of program quality and performance standards which 
specify acceptable levels of performance. The statute requires that these standards measure "the 
increased employment and earnings of participants and the reduction in welfare dependency" 
(Section 106) reflect the volume and quality of placements and to demonstrate cost- 
effectiveness. 5 

Based on these considerations, the U. S. Department of Labor (DOL) adopted the 
following core performance standards during the first year of JTPA's implementation: 

• Entered employment rate (reported separately for youth, adults and welfare 
populations); 

• Cost per entered employment (reported separately for youth and adults); 

• Average hourly wage at placement (for adults); and 

• Youth positive termination rate. 

To obtain information on longer-term outcomes, DOL later added five additional 
standards measuring placements and wages after 90 days. Governors were allowed to select eight 
of the twelve standards to report for their SDAs. 

DOL sets national numeric goals for the performance measures to monitor program 
outcomes in each state. States may adjust their outcomes based on a regression model to account 
for participant characteristics and local labor market conditions. These adjustments may be 
applied on a statewide or individual project basis. 

PICs normally develop performance-based contracts with service providers that specify 
numerical goals reflecting the performance standards the SDA must meet. SDAs that exceed 
performance standards at a level determined by the state are eligible to receive additional funding 
from the state's six percent setaside funds established for this purpose and for technical assistance. 
Policies for distributing these funds are established by the states. States may also use their 
incentive funds to provide technical assistance to SDAs that fail to meet performance targets. 

JTPA requires all SDAs to have a management information system (MIS) to record 
performance standards, participant characteristics, financial data and other programmatic data. 
SDAs report this information annually to the state using a standard report format, the JTPA 
Annual Status Report (JASR). States annually reports JASR data to DOL, which uses the 
information to set numeric goals for the performance standards for the next program year. The 
state may also use JASR data to make adjustments to performance standards statewide or for 
individual SDAs. 



5 DOL also wanted to use the standards to help restore credibility to Federal employment 
programs which had been eroded in the private sector by JTPA's predecessor, the Comprehensive 
Employment and Training Act. 
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Impact of Performance Standards 



Performance standards have been a controversial component of the JTPA program. 
During the first few years of JTPA's existence, states criticized DOL for not providing sufficient 
guidance and technical assistance on how to use the standards and the adjustment model. 

The most serious complaint about performance standards, however, was that they caused 
JTPA programs to "cream" participants-that is, serve the more job-ready-and neglect populations 
most in need of training. Since the standards emphasized quick job placement at low cost, 
standards were also criticized for causing SDAs to emphasize short-term training and job search 
activities and to avoid more costly, intensive training. 

In response to these criticisms, the Nation Commission for Employment Policy (NCEP, 
1988) conducted a national study on the impact of performance standards and the U.S. General 
Accounting Office (GAO, 1989) assessed the JTPA participant population. The NCEP study 
focused on the unintended effects of performance standards on clients, service and cost. The 
study found that performance standards did have unintended effects of reducing service to the 
hard-to-serve and reducing the intensity of training. However, the effects were not large and 
there was considerable variation among SDAs, The effects were worse in SDAs that lacked 
strong client or employer policy goals. State and local policies that emphasized exceeding the 
standards and placed greater weight on the cost per entered employment standard had the 
greatest impact on reducing service to the hard-to-serve. The study also found that factors that 
promoted more intensive training were placing emphasis on the wage standard, use of the 
adjustment procedure and specific state and PIC policies that emphasized the hard-to-serve. 

The GAO study examined the types of participants served by JTPA and the training 
offered. The study found that while JTPA served a wide range of participants, the less job-ready 
were underserved and services were not targeted to the less job-ready. In addition, the intensity 
of job training services was less than expected, as more than half of participants received either 
low skill or nonoccupational training or placement assistance only. 

To address these concerns, DOL replaced the initial standards in program year 1990 to 
reflect policy that emphasized longer, more intensive training that leads to long-term employability 
and increased earnings and emphasized training to hard-to-serve populations. The currently 
required standards are: 

• Employment rate at follow-up (reported separately for adults and welfare adults); 

• Weekly earnings at follow-up (reported separately for adults and welfare adults); 

• Youth entered employment rate; and 

• Youth employability enhancement rate. 

The follow-up period is three months after program termination. Most SDAs use third 
party contractors to contact former participants by mail and telephone to obtain the information. 
Some SDAs use unemployment insurance records to locate former participants. As the standards 
only took effect in July 1990, there is not yet information on whether they are having the 
intended effect of promoting more intensive training or any unintended effects. 
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JTPA's experience with the initial performance standards illustrates the unintended 
effects on program design and enrollment decisions that can result if standards emphasize quick 
results. Without clear policy guidance to the contrary, programs may avoid longer-term services in 
favor of less intensive service with faster results. Programs may also be more selective in 
enrollment decisions, giving preference to participants needing less remediation and training. 

The Vocational Education Act 

Vocational education services are supported at the Federal level through the Carl D. 
Perkins Vocational and Applied Technology Act. Federal funds supplement state and local funds, 
which provide the bulk of the financial support for secondary, postsecondary and adult vocational 
education programs. The Perkins Act emphasizes vocational education services for special 
populations such as the handicapped, the economically disadvantaged, disabled, and those with a 
limited proficiency in English. Three-quarters of Federal funds are distributed to local education 
agencies, area vocational-technical schools and postsecondary institutions. The remaining funds 
are used for state administration, state leadership, corrections, and equity programs. 

Vocational Education Standards 

During legislative debate on the reauthorization of the Perkins Act in 1990, calls for 
vocational education performance standards were heard from the National Assessment of 
Vocational Education (NAVE) and the National Association of State Directors of Vocational 
Education (NASDVE). NAVE recommended that state performance standards include 
information on academic achievement, vocational attainment and occupational skills, employment 
outcomes, and the continuity of student training between secondary and postsecondary. 

NASDVE recommended that states choose at least three of the following indicators: 
percent of class completers; student demonstrated competencies at an industry standard level; 
percent of class placement in an occupation related to the field of preparation; percent of class 
placement in related field; percent of class placement in non-related field; average per placement 
hourly wage; and percent of class enrolled in additional learning. 

Congress incorporated these recommendations into Title I, Part B, Section 115 of the 
Perkins Act by requiring all states to develop "statewide standards and measures of performance" 
within two years of enactment for all programs authorized by the Act. The Act requires that state 
developed performance standards should include: 

• Measures of learning and competency gains, including student progress in the 
achievement of basic and more advanced academic skills. 

• One or more measures of performance, including: competency attainment, job or 
work skill attainment; retention or completion of secondary school or its 
equivalent; and placement into additional training or education, military service, or 
employment. 
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• Incentives or adjustments that are designed to encourage services to special 
populations and when appropriate for each student consistent with an 
individualized education plan developed under the Education of the Handicapped 
Act. 

• Procedures for using existing resources and methods developed under ether 
programs receiving Federal assistance. 

In addition, the vocational education standards developed by states are to take into consideration 
the standards and measures developed for JOBS and JTPA. 

Each state is required to develop standards by 1992 in consultation with a Committee of 
Practitioners representing schools, organized labor, business, school superintendents, community 
based organizations and PICs. The Perkins Acts also requires states to include adjustments or 
incentives along with the standards to encourage services to special populations or targeted 
groups. States must also take into consideration the JTPA and JOBS standards when developing 
standards. 

State Activities 

States appear to be having considerable difficulty in developing vocational education 
performance standards (Vocational Training News, 1991), Much of the problem stems from state 
and local programs 9 inexperience evaluating vocational education and lack of a consistent 
framework and objectives. To address this problem, the National Center of Research in 
Vocational Education (NCRVE), in cooperation with the U.S. Department of Education (ED) is 
providing technical assistance to states in developing vocational education indicators. NCRVE is 
working with states in the development of indicators individually and through workshops. 

The NCRVE reported that 24 states had never conducted an assessment of their 
vocational education programs and were therefore starting from scratch in developing vocational 
education indicators. As of November 1991, 20 states had not yet determined which performance 
indicators to use (Vocational Training News, 1991). Of the 30 states that have identified 
performance indicators, the majority report that separate indicators are being used for secondary 
and postsecondary vocational programs and that separate measures will be used for special 
populations. 

There currently exists little specific information on state of state activities related to 
developing vocational education performance indicators, although NCRVE will soon release a 
report on this topic. The activities of of four states-Florida, Illinois, Minnesota and Missouri-are 
summarized below from the research literature (Asche, 1990; Apling, 1989). 

♦ Florida -Since 1985-86 the state has used placement rates into employment or 
other training programs in its vocational education funding decisions. Data on 
placement rates have been obtained from program evaluations. Programs with 
placement rates of less than 60 percent must be reviewed by state staff and 
programs are ineligible for funds if their placement rate is less than 70 percent for 
three consecutive years. 
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• Illinois -The state is pilot testing in local programs the use of i>ix vocational 
education indicators: job placement, enrollment in other education and training 
programs, employer satisfaction, student satisfaction, employability skills 
attainment, and cost. 

• Minnesota- Four vocational education indicators are being considered: number of 
employed graduates, number of graduates employed in occupations related to the 
program, employer's satisfaction with the quality of the graduate's work, and 
program costs. 

• Missouri -Since 1986-87 the state has used two sets of measures to distribute 
incentive funds to vocational-technical schools. These measures are: job or 
continuing education placement of program completers and labor market supply 
and demand for a given skill area. 

Impact of Standards on the Program 

States are required to evaluate all vocational education secondary and postsecondary 
programs based on the standards that are being developed. It is too early to assess the impact of 
standards on the quality and operation of vocational education programs. Reports prepared prior 
to enactment of the Perkins amendments discuss the possible impacts of standards on vocational 
education programs. For example, Apling (1989) has suggested, based on the JTPA experience, 
that standards might reduce services to those most in need unless they are carefully designed and 
implemented. A related concern is that programs might avoid effective longer-term services in 
favor of less expensive services that produce some quicker results. 

Other cautions about potential vocational education indicators relate to concerns that 
indicators typically measure quantity, not quality; provide information for policymakers rather than 
educators; and emphasize aspects of programs that are easily measured areas rather than program 
elements that may have a stronger effect on quality (Asche, 1990). Finally, the difficulty states 
are experiencing defining standards points to the need for clear guidance and a framework prior 
to the development of indicators. 



Food Stamp Employment and Training Program 

The Food Security Act of 1985, as amended by the Hunger Prevention Act of 1988 
authorizes the Food Stamp Employment and Training Program. This program, operated by the 
U.S. Department of Agriculture, is designed to provide education and job training for Food 
Stamp recipients between the ages of 17 and 59 to decrease their reliance on Food Stamps. 

Food Stamp Indicators 

Through 1991 FSETP had a single participation requirement standard: states were 
required to place 50 percent of program participants in an employment and training program. 
Beginning in 1992 the placement rate standard was reduced to 10 percent of participants with a 
goal of having the program provide substantive services and not provide referral services. 
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FSETP's authorizing legislation required the development of additional outcome indicators 
of program quality. A national advisory committee was established to review various program 
evaluations and recommend appropriate quality indicators. Four indicators were recommended by 
the advisory committee: (1) employment entry; (2) wage rate increases; (3) increased educational 
levels of family members; and (4) food stamp case closure. Standards were also established for 
each of these indicators. Three standards relating to increased educational levels are: (1) passing 
a state-approved test such as the GED; (2) participation in state literacy activities; and 
(3) completion of 64 hours of an educational activity. 

Passage of these requirements resulted in an outcry of concern by interested parties, 
especially the American Public Welfare Association and State Food Stamp Directors. They were 
especially concerned that the collection of data related to these indicators and standards would 
require a significant investment in the development and operation of management information 
systems (MIS) and a substantial time commitment by local program staff, neither of which was 
feasible in view of limited program resources. 

In response to these concerns, the Congress enacted legislation in 1991 (P.L. 102-237) 
which postponed enactment of quality indicators and standards until after the JOBS program 
recommends standards in October 1993. The statute further suggested that quality indicators and 
standards for the Food Stamp Employment Program should consider JOBS quality standards. 

Impact of FSETP Standards 

While the impact of FSETP standards cannot yet be addressed, the initial process standard 
of 50 percent placement into employment program demonstrates that success in meeting a 
performance standard does not necessarily lead to the accomplishment of the larger program goal. 
Local programs were able to place participants, but not necessarily employ them or make them 
more self-sufficient. The quality of the programs to which participants were placed and their 
ability to find jobs for participants were not necessarily considered. 

The failure of the field to adopt the standards that the national committee developed 
demonstrates the importance of local input into the development of standards and the need to 
consider data collection capabilities of local programs as part of the development process. 



Job Opportunities Basic Skills Program 

The centerpiece of the Family Support Act of 1988, which overhauled the nation's welfare 
system, is the JOBS program. JOBS seeks to promote the self-sufficiency of AFDC recipients by 
requiring states to provide education, training, employment services, and support services. AFDC 
recipients who do not have a high school diploma and who cannot demonstrate literacy skills or 
meet certain conditions must enroll in high school or basic education programs. Other 
participants must receive employment-related training. 

JOBS Performance Indicators 

Since its inception, JOBS has had a requirement that programs meet statutory provisions 
tor participation and local funding of the program. States must spend at least 55 percent of JOBS 
funds on target groups and participation must average at least 20 hours per week. In addition, 
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seven percent of the state's Aid Tor Families with Dependent Children (AFDC) caseload must 
participate in JOBS. This figure is to be raised incrementally to 20 percent of the caseload by 
1995. 

The Family Support Act requires the Department of Health and Human Services (HHS) 
to develop performance standards for JOBS programs that are consistent with those of JTPA. 
The legislation notes that possible program indicators should include: increased employment, self- 
sufficiency, reduced welfare dependence, and long-term job retention. Information obtained 
through an evaluation of the JOBS program will be used in suggesting possible program 
indicators. HHS also will consult with experts before performance standards are recommended to 
the Confess. 

Little has been done to date to establish performance standards, since JOBS oniy became 
operational in October 1990 and is not yet fully implemented in all states. HHS has established 
an advisory panel for evaluating the JOBS program that will make recommendations on the 
development of performance standards. 

Impact of JOBS Standards 

Since the formal performance standards for the JOBS program will not be established for 
almost two years, it is premature to discuss their impact on the JOBS program. However, the 
impact of the participation rates included in the JOBS statute provide some insights suitable for 
adult education. 

States not meeting the required participation rates lose the possibility of receiving 
additional Federal funds. Participation rates established by the JOBS authorizing legislation could 
mean that states concentrate on providing less expensive services which might not meet the long- 
term needs of participants in order for additional people to receive services. A similar concern 
was discussed in the section of this paper that examined vocational education indicators of 
program quality. 

Performance Indicators and Adult Education Programs 

As the Federal Government and states continue their efforts to develop indicators of 
program quality, the experiences of JTPA, vocational education, FSETP, and JOBS will be useful 
to consider, even though adult education typically involves a much broader range of goals than 
these other programs. Such goals include helping participants achieve literacy, obtain a high 
school equivalency credential or improve self-confidence, in addition to employment-related 
outcomes. Since JTPA has the most extensive experience with performance indicators, it provides 
the most guidance for adult education. JTPA has also developed performance standards in 
conjunction with indicators and this experience should also be useful for the adult education field. 
Although states are only now in the process of developing vocational education indicators, their 
experiences also provide some insights that will also be useful for the adult education field. 

Based on the experience of these other programs with performance indicators, the 
following lessons appear most relevant to adult education. 
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• Establish appropriate indicators and standards; 

• Allow state and local flexibility; 

• Develop management information systems; 

• Provide states with technical assistance; and 

• Take steps to avoid unintended effects of indicators. 

Establish Appropriate Indicators and Standards 

The adult education statutory requirement for performance indicators provides only 
minimal guidance to the states which are responsible for their design and development. 
Experiences from vocational education provide important insights for ED and the states in 
developing quality indicators for the adult education program. Asche (1990) recommends that 
states adopt the following plan for systematic development of indicators for vocational education: 

• Develop indicators as part of a system for evaluation, planning, and improvement, 
rather than in isolation; 

• Determine the validity of the indicator, specific data requirements to measure the 
indicator, and the indicator's usefulness for improving the program; and 

• Initially adopt a minimum set of indicators. 

Other research points out that "each successful performance indicator must be endorsed 
by vocational-technical education's advocates and adversaries alike 11 (Stevens, 1992). Stevens 
(1992) suggests four criteria for states to select performance indicators: representativeness, 
accuracy of measurement, time dependence, and appropriateness of available unit of analysis. 
Performance indicators, Hoachlander (1990) suggests, should be: clear and precise; quantifiable; 
easy and inexpensive to measure; timely; and consistent from program to program and across time. 

Allow State and Local Flexibility 

The JTPA experience indicates that allowing state and local flexibility in setting 
performance standards is an important element of successful implementation of standards. JTPA 
permits local flexibility (1) in defining some standards and (2) adjusting the numeric level of the 
standard to reflect local conditions. 

The manner in which other programs have allowed indicators to be defined at the local 
level is of particular interest to the adult education field because of the extremely decentralized 
nature of AEA services and the diversity of possible objectives for participants. The experiences 
of JTPA and vocational education in providing states and localities with the responsibility for 
establishing indicators at the state and local levels is consistent with the philosophy of adult 
education programs which allows for great flexibility in setting goals for participants. 

Of the sue core JTPA performance standards, DOL allows the localities (i.e., the PICS) to 
define how to measure the youth employability enhancement rate. This rate measures the 
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attainment of skills by participants under 18 that will enhance their employability or earning 
potential The local SDA must define three elements of the standard: (1) basic skill and 
occupational competencies, (2) satisfactory progress, and (3) PIC recognized competencies. SDAs 
have defined these elements as follows. 

♦ Basic skills competency. These skills reflect educational attainment and SDAs 
usually establish skill areas and benchmarks by consulting with local school boards. 
Academic areas such as reading, writing, mathematics or language arts are typically 
selected. Benchmarks may include improvement in tests scores or in grade point 
average. 

♦ Occupational skill competency . Occupational skills are specific skills required for a 
certain job* Spates and local PICs define these skills according to their labor 
market needs. Examples of these skills are familiarity with equipment and/or 
speed using equipment. 

♦ Satisfactory progress . To make satisfactory progress, JTPA participants must 
improve one step from where they were at enrollment either in the area of 
education, employment, or employment readiness. For instance, a participant may 
complete the next major education level by obtaining either a GED, Associate 
degree, or Bachelor degree. Alternatively, a participant may acquire an 
occupational skill competency* 

♦ PIC recognized competencies . The PIC establishes the basic and occupational 
competencies. In addition, it may define further work maturity and pre- 
employment skills that may be used as performance standards. These skills help 
the participant obtain a job and behave professionally on the job. Examples of 
these skills are resume writing, discipline, punctuality and proper dress. 

Procedures for establishing basic, occupational and other PIC-recognized competencies 
vary considerably at the local level. The process is time-consuming and requires collaboration 
with local school boards and employers, as well as an understanding of the local labor market 
conditions. Specific skills and definitions vary across SDAs. Exhibit 1 presents a summary of the 
general procedures used by PICs. 

The performance standard adjustment model allows states to adjust performance standards 
to local SDA conditions. The model uses a regression procedure to make standards more 
sensitive to local wages, unemployment levels and characteristics of service populations. Without 
local adjustment, performance goals will be unrealistic and adversely affect decisions of program 
design and targeting of service populations. 

Within SDAs, performance standards also can be adjusted for local training providers. 
Most SDAs contract for training services through performance-based contracting. Providers are 
paid after meeting specific goals or performance standards. These goals or standards also may be 
adjusted according to the service population or training approach of the provider. Payments may 
be tied to other factors, such as who is served. Without appropriate adjustment, the standards 
will influence program design and participant characteristics inappropriately. Additional evidence 
has been cited by Joseph Wholey (1990) that adjusted standards can improve the correlation 
between program outcome measures and program impacts. Adjusted standards also appear to 
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EXHIBIT 1 

PIC COMPETENCY SELECTION AND DEFINITION PROCESS 



STEP 1A 

IB 
1C 


The local PIC and SDA staff examine the needs of the local labor 
market and generate a list of potential skills and their respective 
uciiniuuns. 

The PIC surveys local employers to allow them to comment on the 
competency skills and definitions. 

The results of the survey are used to refine, revise, and change the list 
of potential skills and definitions and produce a final list. 


STEP 2 


The PIC and SDA staff members then set the level of each skill 
participants must reach to fulfill the performance standard. 


STEP 3 


The PIC and SDA staff members devise a plan for assessing the 
competencies. A mixed set of assessment measures are used. Actual 
performance, performance assessment, and standardized tests represent 
some common methods used when assessing the competencies. 


STEP 4 


Instructional design is the next decision made by the PIC and SDA 
staff. They try to create an instructional system which is based on the 
competencies. 
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create more appropriate incentives for program operators by taking explicit account of 
participants' individual differences. 

SEAs may need to adjust quality indicators for individual LEAs in the adult education 
program for the same reasons. The incoming skill levels of students, resources available and 
target populations will vary across LEAs and indicators will need to be sensitive to these 
differences. Similarly, LEAs will need to adjust standards for providers within the LEA 
depending on who is served and the provider's approach. A program serving an institutionalized 
population, for example, should be held to different standards than a GED program. 

Develop a Management Information System 

JTPA programs have an automated MIS with a standardized reporting form (JASR) that 
all SDAs must use. The form and MIS allow for collection of a common set of data, minimize 
burden and facilitate analysis. Adult education programs have no corresponding data collection 
system or MIS. The Annual Performance Report currently completed by states for ED does not 
include quality indicators. ED and the states may need to develop new data collection systems to 
collect indicator data. 

A computerized MIS greatly assists the support, development, and operation of 
performance indicators and standards. An MIS permits an efficient method for standardizing, 
collecting, and reporting the data needed to determine whether a programs standards have been 
met and can provide data for program evaluation purposes. A coordinated system of indicators, 
data collection, and evaluation provides quality data, minimizes data collection burden for states 
and local projects, and prevents duplicate data collection efforts. 

It is important, however, that whatever data reporting requirements are established, they 
not be excessively burdensome. A three year indicator development process in the Food Stamp 
Employment and Training Program was negated because of what were seen as excessive and time- 
consuming reporting requirements for implementing the indicators. The adult education field 
should ensure that the indicators are clear and measurable so that the data collection 
requirements do not result in an undue burden on states and localities. 

Provide Technical Assistance to States 

Statutes requiring states to develop indicators of program quality often provide only vague 
guidelines about the specific indicators that are to be established. The adult education statutory 
requirement, for example, offers three examples of indicators: recruitment, retention, and 
improvements in participants' literacy skills. Programs such as vocational education and adult 
education also have limited experience using quantifiable outcome measures to assess program 
quality. Only a few states, such as Arkansas, Tennessee, and Massachusetts, are experienced in 
developing explicit indicators for adult education (Condelli, Koloski, and Webb, 1992). 

For these reasons technical assistance in developing performance indicators for adult 
education should be available to states. ED is offering technical assistance to states developing 
vocational education standards through the National Center on Research for Vocational 
Education. This assistance is being provided through workshops, written materials, and on an 
individual basis. 
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Areas of technical assistance should include information about the purposes, uses, and 
interpretation of quality indicators. Technical assistance should cover topics such as how to adjust 
indicators to match local conditions, the policy reasons for the use and emphasis on particular 
indicators and programmatic implications of using indicators. Such assistance should minimize the 
chances that indicators will have unintended effects on program services and help ensure that the 
use of indicators will lead to improvements in program quality. 

Take Steps to Avoid Unintended Effects of Indicators 

A serious concern about using performance indicators and standards relates to the possible 
unintended effects they may have on the mix of program services offered and characteristics of 
participants. Specifically there is concern that programs will choose to offer less expensive and 
shorter-term services so that more participants can be served. Another possible unintended 
impact of performance indicators involves the "creaming" of participants, by not providing services 
to those most in need, so that larger improvements in performance can be achieved by the 
program. 

JTPA offers practical experiences relating to unintended effects of indicators on program 
services and participant characteristics. Performance standards often led SDAs to give lower 
priority to the less job-ready participants and to emphasize less intensive job-training. However, 
the NCEP analysis revealed that unintended effects are not inevitable. Two factors exacerbated 
the negative effects: lack of clear policy and goals for the SDA and overemphasis on standards, 
especially the cost standard. From this analysis four methods for minimizing or avoiding 
unintended effects can be identified: 

• Clear expressions of Federal policy on how the standards relate to policy goals; 

• Ongoing input and consultation from local providers on the implementation and 
impact of standards; 

• Setting standards at an appropriate level; and 

• Emphasizing and providing incentives for meeting standards that reflect policy- 
relevant goals. 

When program policy is clearly stated it can guide program design. Performance standards 
can then be established to measure progress toward policy goals. When used in this way, 
performance standards can be an effective method for monitoring progress toward policy 
objectives. The standards operationalize policy by setting targets and a common set of rules for 
local programs. 

Performance standards become problematic and likely to produce unintended effects when 
there is an absence of policy or when policy is poorly defined. In this situation, the standards fill 
the policy vacuum and become ends in themselves. Instead of using performance standards to 
gauge progress toward meeting policy objectives, the process is reversed: achievement of 
performance goals guides program design. The NCEP study revealed that overemphasis on 
performance standards produced similar problems. SDAs lost sight of policy objectives when 
standards were set too high or when incentives for meeting the standards were set too stringently. 
In this situation standards unduly influenced program design decisions. 
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Due to the highly decentralized nature of the JTPA system, policy at the Federal level is 
not sufficient to prevent unintended effects. Since states and SDAs have control over local 
programs (and their own policies), Federal policy does not always filter down to the local level. 
Consequently, technical assistance and frequent communication among all levels is necessary to 
achieve policy consistency and interpretation of policy at the local level. Ongoing consultation 
and input from local providers permits assessment of whether standards are appropriate, are being 
used to reflect policy, and identification of adjustments and changes that may be needed. Adult 
education programs need also to be aware of this need, since this delivery system is similarly 
decentralized-from ED to state and local education agencies. 

The JTPA experience with the "cost per entered employment" standard illustrates how 
lack of policy guidance and overemphasis can lead to unintended effects. DOL established this 
standard to meet JTPA requirements and to allow states, SDAs and DOL to monitor the cost 
efficiency of training programs. However, when not contradicted by state or local training policy, 
the standard was interpreted in some SDAs to mean that inexpensive training programs were 
preferred. In other areas, incentives were strongly tied to program costs, leading to an 
overemphasis on that standard. The result was that the program costs unduly influenced program 
design decisions and lead to short-term, low- intensity training programs in many SDAs. 
Consequently, the more job-ready, who could be trained quickly and inexpensively, were more 
likely to be served in many areas-leading to the "creaming" problem. 

A standard can also be overemphasized when the performance level is set too high A 
natural tendency is to set performance standards to a high numeric level on the belief that higher 
program quality will result. This approach is often misguided. The JTPA experience indicates 
SDAs were able to achieve standards no matter how high they were set. FSETP programs also 
were able to meet their high participation standards. However, overly high standards affect 
enrollment and program design decisions, leading to creaming and low intensity services. The 
JOBS program may also be experiencing this problem, as many states believe the 20 hour 
participation rule is too high (GAO, 1991). Standards should be set at a moderate level, enough 
to achieve policy objectives but not so high as to trigger undesirable effects. 

The NCEP study also illustrates how performance standards can be used positively-to 
promote more desirable outcomes. The study found that emphasis on the wage standard was 
related to more intensive training approaches. Since higher wage jobs typically require greater 
skill levels and thus more training, this finding is not surprising. However, it demonstrates how a 
desired policy outcome-creating workers with higher skills-can be achieved through the use of 
performance standards. A second factor that promoted intensive training design was adjusting the 
standards to local conditions, as discussed above. 

Another way to use performance standards in a positive way is to use incentives to 
emphasize standards that meet policy-relevant go^ls. JTPA requires six percent of each state's 
basic allocation to be set aside for incentive payments to SDAs for exceeding performance 
standards. The governor decides how incentive funds are used and paid out and may tie 
incentives to individual standards that reflect state employment policy goals. SDAs are thus 
rewarded for exceeding standards with additional funds, not punished for failing to meet standards 
through withholding of funds. The incentives not only allow the state to emphasize particular 
standards, but create intra-state competition among SDAs that has been beneficial in promoting 
the quality of JTPA programs. 
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